Skip to content

[torch-bisect] fix repro cmdline bug and enhance with more cases - #172

Merged
xuzhao9 merged 10 commits into
mainfrom
xz9/fix-bisect
Apr 16, 2026
Merged

[torch-bisect] fix repro cmdline bug and enhance with more cases#172
xuzhao9 merged 10 commits into
mainfrom
xz9/fix-bisect

Conversation

@xuzhao9

@xuzhao9 xuzhao9 commented Apr 14, 2026

Copy link
Copy Markdown
Contributor
Last good commit: 9d49044
First bad commit: 34cdf49
Metric: geomean_speedup
Threshold: 0.95
Rel_tol: 0.001
Dtype: amp
Arch: b200
Device: cuda
Suite: timm_models
Compiler: cudagraphs
Mode: training
Branch: main
Repo: pytorch/pytorch
Model: inception_v3
Regression: 3.59x -> 3.33x (7.5%)
REPRO_CMD: python benchmarks/dynamo/timm_models.py --performance --amp --training --only inception_v3 --inductor

Test workflow:
https://github.com/pytorch/pytorch-integration-testing/actions/runs/24430996449

The workflow works, however, the regression could not be reproduced:

No regression detected: current value 3.881, 3.888 / 3.881 - 1 == 0.1803658850811677%, threshold 4.0%)

Regression report 2:

Between commits 7ba94e7f (Apr 12) and 437c81a2 (Apr 13), 20+ torchbench models dropped 13-22% speedup on H100 bfloat16 inference:
nanogpt: 8.67x to 6.72x (-22%)
phlippe_resnet: 5.63x to 4.51x (-20%)
squeezenet1_1: 4.20x to 3.39x (-19%)
mobilenet_v3_large: 5.80x to 4.78x (-18%)

Command:

python benchmarks/dynamo/torchbench.py --performance --bfloat16 --inference --only mobilenet_v3_large --inductor
python benchmarks/dynamo/torchbench.py --performance --bfloat16 --inference --only nanogpt --inductor

Test workflow:

(h100, mobilenet_v3_large)
https://github.com/pytorch/pytorch-integration-testing/actions/runs/24477745032

(h100, nanogpt)
https://github.com/pytorch/pytorch-integration-testing/actions/runs/24481090032

@meta-cla meta-cla Bot added the cla signed label Apr 14, 2026
@xuzhao9 xuzhao9 changed the title [bisect] fix repro cmdline bug [wip][bisect] fix repro cmdline bug Apr 14, 2026
@xuzhao9 xuzhao9 changed the title [wip][bisect] fix repro cmdline bug [torch-bisect] fix repro cmdline bug Apr 15, 2026
@xuzhao9
xuzhao9 requested a review from huydhn April 15, 2026 01:59
@xuzhao9 xuzhao9 changed the title [torch-bisect] fix repro cmdline bug [wip][torch-bisect] fix repro cmdline bug Apr 15, 2026
@xuzhao9 xuzhao9 changed the title [wip][torch-bisect] fix repro cmdline bug [torch-bisect] fix repro cmdline bug Apr 16, 2026
@xuzhao9 xuzhao9 changed the title [torch-bisect] fix repro cmdline bug [torch-bisect] fix repro cmdline bug and enhance with more cases Apr 16, 2026

@huydhn huydhn left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@xuzhao9
xuzhao9 merged commit 177864d into main Apr 16, 2026
3 of 8 checks passed
@xuzhao9
xuzhao9 deleted the xz9/fix-bisect branch April 16, 2026 13:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants